Simultaneous translation of open domain lectures and speeches

ABSTRACT

A real-time open domain speech translation system for simultaneous translation of a spoken presentation that is a spoken monologue comprising one of a lecture, a speech, a presentation, a colloquium, and a seminar. The system includes an automatic speech recognition unit configured for accepting sound comprising the spoken presentation in a first language and for continuously creating word hypotheses, and a machine translation unit that receives the hypotheses, wherein the machine translation unit outputs a translation, into a second language, from the spoken presentation.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims priority to U.S. Provisional PatentApplication No. 60/854,586 filed Oct. 26, 2006.

BACKGROUND

Growing international information strictures and decreasing travel costscould make the dissemination of knowledge in this globalized worldeasier if language banners could be overcome. Lectures are a veryeffective method of knowledge dissemination. Such personalized talks arethe preferred method because they allow the speakers to tailor theirpresentation toward a specific audience, and in return allow thelisteners to get the most relevant information through interaction withthe speaker. In addition, personal communication fosters the exchange ofideas, allows for collaboration, and forms ties between distant units,e.g. scientific laboratories or companies. At the same time it isdesirable to allow the presenters of talks and lectures to speak intheir native language because, no matter how proficient in a foreignlanguage, one will always feel more confident speaking in their nativetongue. To overcome this obstacle human translators are currently theonly solution. Unfortunately, translation services are oftenprohibitively expensive such that many lectures are not given at all asa result of language barriers. Thus, there is a need for systems andmethods that use machine translation techniques to provide translationservices at no or low cost to a wide audience, making it possible toovercome language barriers and bring people closer together.

SUMMARY

In various embodiments, the present invention is directed to a real-timeopen domain speech translation system for simultaneous translation of aspoken presentation that is a spoken monologue comprising one of alecture, a speech, a presentation, a colloquium, and a seminar. Thesystem includes an automatic speech recognition unit configured foraccepting sound comprising the spoken presentation in a first languageand for continuously creating word hypotheses, and a machine translationunit that receives the hypotheses, wherein the machine translation unitoutputs a translation, into a second language, from the spokenpresentation.

In various embodiments, the present invention is directed to a method ofreal-time simultaneous translation of an open domain spoken presentationthat is a spoken monologue comprising at least one of a lecture, aspeech, a presentation, a colloquium, and a seminar. The method includesrecognizing speech of the spoken presentation, and translating therecognized speech into a second language.

In various embodiments, the present invention is directed to anapparatus for real-time simultaneous translation of an open domainspoken presentation that is a spoken monologue comprising at least oneof a lecture, a speech, a presentation, a colloquium, and a seminar. Theapparatus includes means for recognizing speech of the spokenpresentation, and means for translating the recognized speech into asecond language.

In various embodiments, the present invention is directed to a computerreadable medium having stored thereon instructions which, when executedby a processor, cause the processor to translate an open domain spokenpresentation that is a spoken monologue comprising at least one of alecture, a speech, a presentation, a colloquium, and a seminar, whereinthe processor:

recognizes speech of the spoken presentation; and

translates the recognized speech into a second language.

Those and other details, objects, and advantages of the presentinvention will become better understood or apparent from the followingdescription and drawings showing embodiments thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate examples of embodiments of theinvention. In such drawings:

FIG. 1 illustrates a schematic overview of an embodiment of asimultaneous translation system 10 and its models;

FIG. 2 illustrates an embodiment of a translation system having variousoutput devices;

FIG. 3 illustrates an open domain process according to an embodiment ofthe present invention.

DESCRIPTION

Various embodiments of the present invention describe methods andapparatuses for open domain speech translation. Embodiments may be usedto simultaneously translate, in real time, unlimited or open domainlectures and other spoken presentations or monologues such as, forexample, parliamentary sessions using automatic speech recognition (ASR)techniques. Embodiments may be used to translate such presentations fromone language into one or more target languages such as, for example,translating lectures on a new domain from English to Spanish and German.In various embodiments, small amounts of lecture data were used toimprove system performance by using language model adaptation withsimilar web data.

Lectures, seminars and presentations of any kind present problems fordomain-unlimited speech translation because of the spontaneity of freespeech, disfluencies, and ill-formed spontaneous natural discourse, thespecialized vocabularies, topics, acronyms, named entities andexpressions in typical lectures and presentations (by definitionspecialized content), the real-time & low-latency requirements, onlineadaptation to achieve simultaneous translation, and selection oftranslatable chunks or segments.

To address these problems in ASR and MT engines, embodiments of thepresent invention speed up recognition by adapting acoustic and languagemodels to individual speakers. Also, in various embodiments the size ofthe acoustic model is restricted and the search space is more rigorouslypruned. Further, in various embodiments, to adapt to a particularspeaking style and domain, the language model is tuned offline on slidesand publications by the speaker, either by reweighting available textcorpora or by retrieving pertinent material on the Internet, similarpublications or previous publications lectures by the same speaker.

Speech translation systems generally consist of two major components:speech recognition and machine translation. Available audio recordingsare recognized and the resulting hypothesis transmitted to the machinetranslation, which outputs the translation Performing speech recognitionon a stream of audio data is generally done on a per utterance level,whereas the utterance boundaries have to be determined with the help ofan audio segmenter in front of the speech recognizer. When the audiodata contains noise artifacts or cross-talk, this strategy can be usefulbecause such phenomenon can be removed in advance, leading to anincrease in ASR performance.

The techniques used in such audio segmenters often require a globaloptimization over the whole audio data and therefore may be infeasiblefor a simultaneous translation system. On the other hand,speech/non-speech based audio segmenters introduce an additionallatency, because the classification of speech/non-speech frames has tobe followed by a smoothing process to remove mis-classifications. Also,in the architecture of the speech recognizer itself, longer utterancesoften lead to a disproportional increase of the search space duringdecoding and therefore to longer decoding times.

Most machine translation systems were developed in the context of texttranslation and have to cope with all the differences between a sourceand target language such as different amount and usage of word ordering,morphology, composita, idioms, and writing style, but also vocabularycoverage. In addition to the differences between a source and targetlanguage, spoken language, i.e. speech, differs also in style While textis mostly grammatically correct, spoken language and especiallyspontaneous or sloppy speech contains many ungrammaticalities such ashesitations, interruptions and repetitions. In addition, the choice ofwords and the amount of vocabulary used differ between text and speech.Another difference is that text is written and semantic boundaries aremarked with punctuation while for speech, either humans or automaticspeech recognition systems have to be used to transcribe the data. Thisintroduces an additional source of error and, due to theungrammaticalities of speech, semantic boundaries are difficult toidentify. This is a problem because traditionally almost all machinetranslation systems are trained on bilingual sentences, so that theinput should be also sentence-like segments, which may be a problem whena low latency speech translation system is required. To cope with thisproblem, a third component focusing on the resegmentation of the ASRhypothesis with the requirement of achieving a high translation qualitytogether with a low latency is described herein according to variousembodiments.

FIG. 1 illustrates a schematic overview of an embodiment of asimultaneous translation system 10 and its models. A speech recognitionunit 12 produces partial hypotheses 14 based on an acoustic model 16, alanguage model 18 and a dictionary 20. The hypotheses 14 are merged,filtered and resegmented by a resegmentation unit 22 using a boundarymodel 24. The processed hypotheses are transferred to a machinetranslation unit 26 for translation into another language. Suchtranslation units 26 are specified by using a translation model 28 and alanguage model 30 and/or dictionaries or grammars. In variousembodiments, one of the systems 10 is used for each desired language.

FIG. 2 illustrates an embodiment of a translation system 40 havingvarious output devices. The system includes a targeted audio device 42,a heads up display unit (glasses or goggles) 44 and a monitor 46 thatdisplays subtitled translations. When an attendee at a spokenpresentation wears the heads tip display unit 44, text in the translatedlanguage is projected into the field of view of the attendee. In variousembodiments, the targeted audio device 42 such as that described in D.Olszewski, F. Prasetyo, and K. Linhard, “Steerable Highly DirectionalAudio Beam Loudspeaker”, in Proc. of the Interspeech, Lisboa, Portugal,September 2006. The targeted audio device is a beam-steered loudspeaker,consisting of several small ultrasound loud-speakers. The device outputsaudio in a beam with a width of about 1-2 meters. People sitting withinthe beam are able to hear the synthesized translation output and peopleoutside the beam do not. In various embodiments, several such targetedaudio devices are assigned in various languages to accommodate eachparticipant in the lecture room.

In various embodiments, the following output devices may be used:Display Screens Naturally, output can be delivered via traditionaldisplay technology: display on separate screens, or as subtitles; butall add distraction and inconvenience and it limits output to onelanguage.

Personalized headphones or PDA screens. This allows for individualchoice of output language (if several are provided), but it isinconvenient to wear.

Translation goggles: Heads-up display goggles that display translationsas captions in a pair of personalized goggles. Such a personalizedvisual output mode exploits the parallelism between acoustic and visualchannels. This is particularly useful if listeners have partialknowledge of a speaker's language and wish to add complementary languageassistance.

Targeted Audio Speakers: A set of ultra-sound speakers with highdirectional characteristics that provide a narrow audio beam toindividual listeners in a small area of the audience, where simultaneoustranslation is required. Since such speakers are only audible in anarrow area, they do not disturb other listeners, and several speakerscan provide different languages to different listeners.

Database: The translated speech may be output to a storage medium suchas a database so that a user can browse, search, index, view, etc. thecontent and its translation at a later time.

In various embodiments, the techniques described in CHIL (Waibel et al.,2004), “Computers in the Human Interaction Loop”, are used for speakerlocalization and tracking, speech activity detection and distant-talkingautomatic speech recognition, without any constraint on the number ordistribution of microphones in the room nor on the number of soundsources active at the same time.

Various embodiments employ the techniques described in TC-STAR (TC-Star,2004), “Technologies and Corpora for Speech-to-Speech-Translation”, forSpeech-to-Speech Translation (SST) for offline applications and forgeneral topics of discussion (not simultaneous lectures), which is acombination of Automatic Speech Recognition (ASR), Spoken LanguageTranslation (SLT) and Text to Speech (TTS).

Various embodiments use loose coupling, passing the first-besthypothesis from the recognizer to the translation component. Translationresults may be reported using the evaluation metrics described in K.Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a Method forAutomatic Evaluation of Machine Translation”. Technical Report RC22176(WO109-022), IBM Research Division, T. J. Watson Research Center, 2002and NIST MT evaluation kit version 11a,http://www.nist.gov/speech/tests/mt, 2004. In various embodiments, allmachine translation (MT) scores are calculated using case-insensitivescoring and one reference translation per test set.

Various embodiments integrate the aforementioned speech recognition andmachine translation systems together with a sentence segmentationcomponent and a speech synthesis into a client-server framework (see C.Füigen, M. Westphal, M. Schneider, T. Schultz, and A. Waibel, “LingWear:A Mobile Tourist Information System”, in Proc. of the Human LanguageTechnology Conf. (HLT), San Diego, Calif., March, NIST, 2001).

Various embodiments use the Janus Recognition Toolkit (JRTk) featuringthe Ibis decoder as described in H. Soltau, F. Metze, C. Fügen, and A.Waibel, “A One Pass-Decoder Based on Polymorphic Linguistic ContextAssignment”, in ASRU, Trento, Italy, 2001. In various embodiments, theSRI Language Modeling Toolkit (SRILM) described in A. Stolcke, SRILM,“An Extensible Language Modeling Toolkit”, in ICSLP, Denver, Colo., USA,2002, is used for language modeling.

In various embodiments, to reach real-time end-to-end performance, thesingle pass speech recognizer is tuned to run faster than real-time byfurther restricting the beam search. In various embodiments, such anarrangement results in an increase in WER to about 13% on the evaluationtalks but improves speed and reduces latency for simultaneoustranslation.

In various embodiments, to keep the latency of the system as short aspossible the speech recognizer starts to decode while the speaker istalking and continuously returns partial back traces with first besthypotheses. Because the machine translation awaits complete sentences asinput, the partial hypotheses are merged together and resegmented tosentence-like segments. Thus, no speech segmentation is performed beforeprocessing by the speech recognizer but is done afterwards so that thesegmentation boundaries may be tuned with respect to optimal machinetranslation performance. In various embodiments, segmentation is done atsilence regions only, whereby additional thresholds are defined toproduce segments with a length of about, for example, five to ten words.Thus, the latency of the system may be limited to, for example, amaximum of about five seconds.

As described herein, for automatic speech recognition (ASR) andstatistical machine translation (SMT) experiments on lectures usingvarious embodiments of the present invention, three different lectureswere selected as development and evaluation data. These lectures wereheld in non-native English by the same speaker on different topics andwere recorded with close talking microphones as in C. Fügen, M. Kolss,D. Bernreuther, M. Paulik, S. Stüker, S. Vogel, and A. Waibel, “OpenDomain Speech Recognition & Translation: Lectures and Speeches”, inICASSP, Toulouse, France, 2006.

Dev: This 24 min talk was held to give a broad overview of currentresearch projects in a lab.

t035: A 35 min talk held as a conference key-note, which is only partlycovered by the Dev talk.t036+: A 31 min talk on the same topic as t035, but held in a differentenvironment and situation.

In various embodiments, for ASR the seminar part of the NIST RT-06Sdevelopment data and the 2006 EPPS development data are used asadditional data sources. For acoustic model training, the followingcorpora are used: ICSI and NIST meeting recordings (see A. Janin, J.Ang, S. Bhagat, R. Dhillon, J. Edwards, N. Morgan, B. Peskin, E.Shriberg, A, Stolcke, C. Wooters, and B. Wrede, “The ICSI MeetingProject: Resources and Research”, in Proc. of the ICASSP MeetingRecognition Workshop, Montreal, Canada, May 2004, NIST and LinguisticData Consortium (LDC), ICSI, ISL and NIST Meeting Speech Corpora at LDChttp://www.ldc.upenn.edu catalog IDs LDC2004S02, LDC2004S05, LDC2004S09,2004), TED lectures (see L. F. Lamel, F. Schiel, A. Fourcin, J. Mariani,and H. Tillmaim, “The Translanguage English Database TED”, in ICSLP,volume LDC2002S04, Yokohama, September 1994), CHIL, seminars (see A.Waibel, H. Steusloff, and R. Stiefelhagen, “CHIL Computers in the HumanInteraction Loop”, in 5th International Workshop on Image Analysis forMultimedia Interactive Services, Lisbon, April 2004,http://chil.server.de), and European Parliament Plenary Sessions (EPPS)(see C. Gollan, M. Bisani, S. Kanthak, R. Schlüter, and H. Ney, “CrossDomain Automatic Transcription on the TC-STAR EPPS Corpus”, ICASSP2005). Table 1 illustrates an overview of the total amount of speech inthe different corpora.

TABLE 1 ICSI NIST TED CHIL EPPS Speakers 463 77 52 67 1894 Duration 72 h13 h 13 h 10 h 80 h

For language model training, some additional text data was used on topof the language model training data of (see C. Fügen, M. Wölfel, J. W.McDonough, S. Ikbal, F. Kraft, K. Laskowski, M. Ostendorf, S. Stüker,and K. Kumatani, “Advances in Lecture Recognition: The ISL, RT-06SEvaluation System”, Interspeech 2006, Pittsburgh, Pa., USA, September2006 and S. Stüker, C. Fügen, R. Hsiao, S. Ikbal, F. Kraft Q. Jin, M.Paulik, M. Raab, Y.-C. Tam, and M. Wölfel, “The ISL TC-STAR Spring 2006ASR Evaluation Systems”, in TC-Star Speech to Speech TranslationWorkshop, Barcelona, Spain, June 2006). Altogether, the followingcorpora were available: talks, text documents from TC-STAR and CHIL,EPPS transcripts, EPPS final text editions, non AMI meeting data (ISL,ICSI, NIST), AMI meeting data, TED lectures, CHIL seminars, broadcastnews data, UN (United Nations) text data released by ELDA, recentproceedings data (2002-2005), web data from UWash (related to ISL, ICSI,and NIST meetings) and web data collected for RT-06S (related to CHILseminars). Table 2 illustrates the amount of words available for eachcorpus. Empty cells indicate that the data was not useful for thatdomain.

TABLE 2 Talks docs eppsS eppsT nAMI AMI TED CHIL BN UN Proc Uwash wCHILWords 93k 192k 750k 33M 1.1M 200k 98k 45k 131M 42M 23M 147M 146M EPPS35% 54% 9% 2% CHIL 15% 8% 0.6% 2.5% 0.8%  24% 12% 15% Dev 36% 1% 12%  3%8%  9% 11% 19%

The acoustic models described hereinbelow were all trained in the sameway, resulting in a size of 16,000 distributions over 4,000 models, witha maximum of 64 Gaussians per model. These models are all based on thesame quint-phone context decision tree and phoneme set that was used forthe RT-06S evaluation system. Furthermore, the acoustic model trainingsetup was taken from the RT-06S system: a first incremental growing ofGaussians; estimation of the global STC transform as in M. J. F. Gales,“Semi-tied covariance matrices” in ICASSP 1998 and a second incrementalgrowing of Gaussians. To train the distributions for the semi-continuoussystem and to compensate for the occasionally worse fixed-statealignments, two iterations of Viterbi training were performed. For theSAT models, four additional iterations of SAT Viterbi training by usingconstrained MLLR in the feature space (FSA) as in M. J. F. Gales,“Maximum Likelihood Linear Transformations for HMM-based SpeechRecognition” Technical report, Cambridge University, Cambridge, UnitedKingdom, 1997 were applied. An MFCC FFT front-end with a 42-dimensionalfeature space after linear discriminant analysis (LDA) and a global STCtransform with utterance-based cepstral mean subtraction (CMS) wereused.

In various embodiments, a single acoustic model is used for bothdomains, EPPS and CHIL seminars and the acoustic model is used on thelecture data. Different acoustic models trained on different subsets ofthe acoustic training material described hereinabove were compared.

A three pass decoding setup is used in various embodiments. The firstpass uses incremental speaker-based vocal tract length normalization(VTLN) and constrained MLLR estimation and is decoded withsemi-continuous models using tight search beams. The second pass usesthe same semi-continuous acoustic models as pass one, but beforedecoding, MLLR (see C. J. Leggetter and P. C. Woodland, “MaximumLikelihood Linear Regression for Speaker Adaptation of ContinuousDensity Hidden Markov Models”, Computer Speech and Language, 9:171185,1995) adaptation together with an estimation of fixed VTLN andconstrained MLLR parameters is performed. For this, the confidenceweighted hypotheses of the previous pass are used. For the third pass,the FSA-SAT acoustic models are used together with the same adaptationscheme applied in pass two. After that, confusion network combination(CNC) (see L. Mangu, E. Brill, and A. Stolcke, “Finding Consensus amongWords: Lattice-based Word Error Minimization”, in EUROSPEECH, 1999) isperformed using the lattices of the third pass only. The same decodingdictionaries and language models are used as for the EPPS and RT-06Sevaluation systems.

In various embodiments, for the CHIL seminars the same language modelsand dictionaries as described in C. Fügen. M. Wölfel, J. W. McDonough,S. Ikbal, F. Kraft, K. Laskowski, M. Ostendorf, S. Stüker, and K.Kumatani, “Advances in Lecture Recognition: The ISL RT-06S EvaluationSystem”, Interspeech 2006, Pittsburgh, Pa., USA, September 2006, areused. The 4-gram language model was trained on AMI and non-AMI meetings,TED, CHIL, data, BN, proceedings and web data related to meetings andCHIL, lectures. The interpolation weights, which were tuned on held-outCHIL, data are shown in Table 2. In one embodiment, the language modelhas a perplexity of 130 on the RT-06S development data, while 16%4-grams, 41% 3-grams, 39% 2-grams, and 4% 1-grams are used. Thedictionary consists of around 59 k pronunciation variants over avocabulary of 52 k. It has an OOV-Rate of 0.65 on the RT-06S developmentdata.

As can be seen in Table 3, acoustic models trained on EPPS alone oradditionally including TED are significant worse than the other twosystems. The performance of the two other systems is nearly identical,which means that adding the EPPS data to the acoustic model trainingdata used in RT-06 (ICSI+NIST+TED) does not hurt (but also does notimprove the overall results).

TABLE 3 CHIL 1st 2nd 3rd cnc EPPS 40.3 .— .— .— TED + EPPS 38.7 .— .— .—ICSI + NIST + TED + EPPS 34.1 27.5 26.2 25.5 ICSI + NIST + TED 34.0 27.126.0 25.5

For the European Parliament Plenary Sessions, the language models anddictionaries as described in Stuker et al. are used. In one embodiment,the 4-gram language model was trained on EPPS transcriptions and finaltext editions, BN, and UN and reached a perplexity of 93 on the 2006EPPS development data, whereas 29% 4-grams, 36% 3-grams, 32% 2-grams,and 4% 1-grams were used. The interpolation weights were tuned on the2005 EPPS development data and are shown in Table 2. The dictionary forEPPS consists of 45 k pronunciations over a vocabulary of 40 k and hasan OOV-Rate of 0.43 on the 2006 EPPS development data.

As can be seen in Table 4 the last system trained without EPPS performsworst Furthermore, compared to the acoustic model used for the 2006 EPPSevaluation (MS23, Stuker et al.), the acoustic model training setupdeveloped for RT-06S is significantly better (MS23 vs. EPPS rows). Anadditional gain can be seen by adding TED, which is also a corpuscontaining European English. By adding the meeting data, the systemimproves not further, instead it is in between the EPPS and TED+EPPSsystems. Nevertheless, after doing confusion network combination, itperforms identical to the TED+EPPS system.

TABLE 4 1st 2nd 3rd cnc MS23 22.6 .— .— .— EPPS 20.8 15.4 14.7 14.5TED + EPPS 20.1 14.8 14.3 14.1 ICSI + NIST + TED + EPPS 20.6 15.1 14.614.1 ICSI + NIST + TED 29.1 .— .— .—

Compared to the CHIL, seminars, the EPPS results compare favorablybecause of the available amount of acoustic and language model in-domaintraining data for EPPS compared to CHIL, where only a very small amountof in-domain data is available. Furthermore, the language used in theEuropean Parliament is more formal and therefore less spontaneous. Thisleads also to a better OOV-rate and language model perplexity with ahigher n-gram coverage for larger n-grams.

Based on the perplexities and OOV-Rates on Dev shown in Table 5 thelanguage model and dictionary built for the CHIL seminars was selectedfor the baseline experiments. This selection holds also for theevaluation talks. The EPPS language model and vocabulary may be, due tothe large amount of in-domain data, too specific. The OOV-rates of theRT-06S (CHIL) vocabulary and for t036+ are low because the talk is notvery specific.

TABLE 5 Dev t035 t036+ PPL OOV PPL OOV PPL OOV CHIL 173 0.22 117 0.27186 0.09 EPPS 205 1.29 230 1.83 229 1.72

As can be seen in Table 6, the acoustic model trained on all dataperforms significantly better than the other models. Thus, this model isused in various embodiments. The baseline results on the lectureevaluation talks are shown in Table 7. With the training setup developedfor RT-06S the results improved compared to the acoustic modelsdeveloped in Fügen and Kolss et al. (MS11 column in Table 7).Furthermore, it can be seen that the system performs well on unseendomains (t035) and different environments (t036+).

TABLE 6 1st 2nd 3rd cnc EPPS 23.9 .— .— .— TED + EPPS 23.4 .— .— .—ICSI + NIST + TED + EPPS 21.4 16.2 15.0 15.5 ICSI + NIST + TED 24.3 .—.— .—

TABLE 7 1st 2nd 3rd cnc MS11 t035 17.3 12.6 12.1 12.2 12.7 t036+ 16.712.0 11.6 11.5 12.4

Lectures are often suitable domains for doing adaptation, because thelecturer and also the topic might be known in advance. Therefore,acoustic and language model adaptation results are given hereinbelow. Ascan be seen, this allows reduction of the decoding setup from three toonly a single decoding pass without any loss in WER.

For acoustic model adaptation an additional amount of around 7 hours ofspeech for the same speaker was available. For adaptation, subsets ofthis data with different durations were used to compute VTLN andconstrained MLLR (FSA) parameters and to perform model based MLLRadaptation. The results can be seen in Table 8. While the adaptationworks well on the evaluation talks, the 7his results are similar tothose achieved after CNC with the baseline systems—the results on theDev talk are worse. This is due to a large channel mismatch between theadaptation material and the Dev talk. To confirm this, the particulartalk itself was adapted and reached for all talks reasonable results(see column sup in Table 8). It can also be seen that doubling theadaptation data results in a relative gain of around 0.5% in WER. Forlanguage model adaptation a first experiment was performed by turningthe interpolation weights and reselecting the different corpora withrespect to the lecture domain. The interpolation weights, tuned on someheld-out data and the selected corpora can be seen in Table 2. Thus, theperplexity on the Dev talk could only be reduced from 173 to 168.Nevertheless there are gains in WER on all lectures, which are reportedin Table 9.

TABLE 8 0.5 hrs 1.5 hrs 3.5 hrs 7 hrs sup Dev 20.9 20.0 19.5 18.9 12.0t035 14.2 13.1 12.6 12.1 10.1 t036+ 13.3 12.3 11.5 10.7 9.3

TABLE 9 unadapted adapted PPL Dev 18.9 16.1 168 t035 12.1 10.5 165 t036+10.7 9.1 193

To find a translation for a source phrase in various embodiments thegeneral word alignment is restricted. Words inside the source phrasealign to words inside the target phrase, and words outside the sourcephrase align outside the target phrase.

The constrained alignment probability is calculated using the IBM1 wordalignment model, but the summation of the target words is restricted tothe appropriate regions in the target sentence. Also, the positionalignment probabilities are adjusted accordingly using Vogel.Optimization is over the target side boundaries i₁ and i₂

$\begin{matrix}{{p_{i_{1},i_{2}}\left( f \middle| e \right)} = {\prod\limits_{j = 1}^{j_{1} - 1}{\sum\limits_{i \notin {({i_{1}\ldots \; i_{2}})}}{\frac{1}{I - k}{p\left( f_{j} \middle| e_{i} \right)} \times {\prod\limits_{j = j_{1}}^{j_{1}}{\sum\limits_{i = i_{1}}^{i_{2}}{\frac{1}{k}{p\left( f_{j} \middle| e_{i} \right)} \times {\prod\limits_{j = {j_{2} + 1}}^{J}{\sum\limits_{i \notin {({i_{1}\ldots \; i_{2}})}}{\frac{1}{I - k}{p\left( f_{j} \middle| e_{i} \right)}}}}}}}}}}} & (1)\end{matrix}$

Similar to p_(i1,i2) (f|e) then p_(i1,i2) (e|f) can be calculated, nowsumming over the source words and multiplying along the target words. Tofind the optimal target phrase the log probabilities are interpolatedand the pair (i₁,i₂) that gives the highest probability is used. Theinterpolation factor c can be estimated on a development test set. Thescores calculated in the phrase alignment are alignment scores for theentire sentence. As phrase translation probabilities the second term inEquation 1 is used.

The beam search decoder combines all model scores to find the besttranslation. In such embodiments, the different models used were: (1)the translation model, i.e. the word-to-word and phrase-to-phrasetranslations extracted from the bilingual corpus according to the newalignment method described herein; (2) a trigram language model—the SRIlanguage model toolkit was used to train the models. (3) A wordreordering model, which assigns higher costs to longer distancereordering. The jump probabilities p(j|j′) of the HMM word alignmentmodel where j is the current position in the source sentence and j isthe previous position was used. (4) Simple word and phrase count models.The former is essentially used to compensate for the tendency of thelanguage model to prefer shorter translations, while the latter can beused to give preference to longer phrases. For each model a scalingfactor can be used to modify the contribution of this model to theoverall score.

The decoding process is organized into two stages: First, theword-to-word and phrase-to-phrase translations and, if available, otherspecific information such as named entity translation tables areinserted into a translation lattice. In the second step, the bestcombinations of the partial translations are found, such that every wordin the source sentence is covered exactly once. This amounts to doing abest path search through the translation lattice, which is extended toallow for word reordering: Decoding proceeds essentially along thesource sentence. At each step, however, the next word or phrase to betranslated may be selected from all words laying or phrases startingwithin a given look-ahead window from the current position.

For training the baseline translation systems, the parallel EPPS corpuswas used. For English-Spanish, a version was created by RWTH Aachenwithin TC-STAR (see Gollan et al.). The English-to-German models weretrained on the EPPS data as provided in P. Koehn, “Europarl: AMultilingual Corpus for Evaluation of Machine Translation”,http://people.csail.mit.edu/koehn/publications/europarl, 2003. Inaddition, a small number of lectures similar in style to development andevaluation data was collected, transcribed, and translated into Spanishand German. Altogether, parallel lecture corpora of about 12,000 wordswere available in each language.

Adapting the MT component of the EPPS translation system towards themore conversational style of lectures was accomplished by a higherweighting of the available lecture data in two different ways. First,for computing the translation models, the small lecture corpora weremultiplied several times and added to the original EPPS training data.

Second, for (target) language model computation, a small tri-gram LM wascomputed on t035 and then interpolated with the original EPPS languagemodel, whereas the interpolation weight was chosen in order to minimizethe perplexity on the development set. In this manner the perplexity onthe Dev talk could be reduced from 645 to 394 for German and from 543 to403 for Spanish. To further adapt the target language models, Spanishand German web data was collected with the help of tools provided by theUniversity of Washington (see University of Washington, web datacollection scriptshttp://ssli.ee.washington.edu/projects/ears/WebData/web datacollection.html, 2006). A small amount of the used search queries werehandwritten, however, most search queries were automatically created byusing the most frequent tri-grams found in the Dev talk. Approximately ¼of all development set ti-grams were used for this. The German andSpanish web corpora collected in this manner consisted out of 175M wordsand 120M words, respectively. The web corpora were again added to theexisting LMs by interpolation, which yielded a perplexity of 200 forGerman and 134 for Spanish. The corresponding perplexities on the t036+talks are 617 and 227, respectively. The effects of translation modeland language model adaptation, as well as the results of the finalsystem, combining both adaptation steps, are shown in Tables 10 and 11for English-to-Spanish and English-to-German, respectively. An elementof better performance is to take advantage of smaller amounts of lecturedata and other text and presentation data in order to improve thelanguage model adaptation.

TABLE 10 Results on manual transcripts are shown in brackets. SystemNIST Bleu baseline (EPPS) 4.71 (5.61) 15.41 (20.54) TM-adaptation 4.78(5.67) 16.05 (21.43) LM-adaptation 5.10 (5.99) 17.58 (22.90) finalsystem 5.22 (6.11) 18.57 (24.00)

TABLE 11 Results on manual transcripts are shown in brackets. SystemNIST Bleu baseline (EPPS) 4.00 (4.71) 09.32 (12.53) TM-adaptation 4.29(5.06) 11.01 (14.95) LM-adaptation 4.37 (5.12) 11.67 (14.96) finalsystem 4.67 (5.47) 13.22 (17.25)

FIG. 3 illustrates an open domain process according to an embodiment ofthe present invention. As can be seen in FIG. 3 several actions may beperformed by the lecture translation system in advance to a lecture. Ifneither the lecturer nor the title or topic of a lecture is known inadvance as determined at startup time, the system loads general speakerindependent acoustic, language, and translation models. Since no furtherinformation is given, the system is also not able to adapt its modelsany further. If only the name of the speaker is known and the speakerhas already given a talk, on which the system has adapted its models andstored in the lecturer database, the system is now able to load speakeradapted acoustic models. Since the topic is unknown it has to loadgeneral or at most slightly adapted language and translation models. Ifthere is no information about the speaker stored in the database,speaker independent models has to be loaded. In both cases theinformation about the speaker can be used to query the internet forprevious lectures or other publications by the same speaker to adapt thelanguage and translation models.

If also the title or even the slides of the talk are available inadvance, this information can be used to search the Internet for evenmore specific material, which can be related to the speaker but also tothe topic. Therefore, topic related keywords or n-grams are extractedfrom the title or slides. Thus, the more information about a speakerthat is shown in advance, the better the models can be adapted and thebetter the system will perform in the end. For performing theadaptation, the material collected from the Internet has first to befiltered and normalized in order to interpolate it with the other moregeneral background models.

While the foregoing has been set forth in considerable detail, it is tobe understood that the drawings and detailed embodiments are presentedfor elucidation and not limitation. Design variations may be made butare within the principles of the invention. Those skilled in the artwill realize that such changes or modifications of the invention orcombinations of elements, variations, equivalents, or improvementstherein are still within the scope of the invention as defined in theappended claims. Also, the examples and experimental results illustratedherein are exemplary and are not intended to limit the scope of theembodiments of the invention.

1. A real-time open domain speech translation system for simultaneoustranslation of a spoken presentation that is a spoken monologuecomprising one of a lecture, a speech, a presentation, a colloquium, anda seminar, the system comprising: an automatic speech recognition unitconfigured for accepting sound comprising the spoken presentation in afirst language and for continuously creating word hypotheses; and amachine translation unit that receives the hypotheses, wherein themachine translation unit outputs a translation, into a second language,from the spoken presentation.
 2. The system of claim 1, wherein thetranslation of the spoken presentation in a second language is output toat least one of a heads up display, a monitor, a directional beam styleaudio output device, a PDA, a screen, a set of headphones, and adatabase.
 3. The system of claim 1, further comprising a resegmentationunit in communication with the automatic speech recognition unit and themachine translation unit.
 4. The system of claim 1, wherein the machinetranslation unit is governed by parameters of at least one of atranslation model, a language model, a word dictionary, a phrasedictionary, and grammar rules.
 5. The system of claim 1, wherein thespeech recognition unit is governed by parameters of at least one of anacoustic model, a language model and a dictionary.
 6. A method ofreal-time simultaneous translation of an open domain spoken presentationthat is a spoken monologue comprising at least one of a lecture, aspeech, a presentation, a colloquium, and a seminar, the methodcomprising: recognizing speech of the spoken presentation; andtranslating the recognized speech into a second language.
 7. The methodof claim 6, further comprising outputting the recognized spokenpresentation.
 8. The method of claim 6, further comprising outputtingthe translated speech.
 9. The method of claim 6, wherein recognizingincludes adapting a recognition unit by modifying one of a languagemodel, an acoustic model, and a dictionary, based on relevantinformation selected from at least one data source comprising at leastone of the Internet, newscoporpora, text books, technical literature,specialist literature, personal publications, and personal documents.10. The method of claim 6, wherein translating includes adapting atranslation unit by modifying one of a language model, a translationmodel, a dictionary, and grammar miles based on relevant informationselected from at least one data source comprising at least one of theInternet, newscoporpora, text books, technical literature, specialistliterature, personal publications, and personal documents.
 11. Themethod of claim 9, where relevant information is selected based onpersonal information comprising at least one of personal profiles,presentation materials, presentation slides, and personal publicationsof a presentation speaker.
 12. The method of claim 10, where relevantinformation is selected based on personal information comprising atleast one of personal profiles, presentation materials, presentationslides, and personal publications of a presentation speaker.
 13. Themethod of claim 6, further comprising merging, filtering, andresegmenting the recognized speech.
 14. The method of claim 8, whereinoutputting includes outputting to at least one of a heads up display, amonitor, a directional beam style audio output device, a PDA, a screen,a set of headphones, and a database.
 15. The method of claim 7, whereinoutputting includes outputting the recognized speech to a database thatis available for browsing, indexing, and searching the recognized speechand translating the speech to the second language.
 16. The method ofclaim 8, wherein outputting includes outputting the translated speech toa database that is available for browsing, indexing, searching viewingand skimming of the translated speech.
 17. An apparatus for real-timesimultaneous translation of an open domain spoken presentation that is aspoken monologue comprising at least one of a lecture, a speech, apresentation, a colloquium, and a seminar, the apparatus comprising:means for recognizing speech of the spoken presentation; and means fortranslating the recognized speech into a second language.
 18. A computerreadable medium having stored thereon instructions which, when executedby a processor, cause the processor to translate an open domain spokenpresentation that is a spoken monologue comprising at least one of alecture, a speech, a presentation, a colloquium, and a seminar, whereinthe processor: recognizes speech of the spoken presentation; andtranslates the recognized speech into a second language.